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Introduction 

Chapter 1 of the Augustus R Hawkins-Robert T. Stafford Elementary and 
Secondary School Act of 1988 made some fundamental changes in the implementation of 
Chapter 1 in local school districts. One of the major changes was the requirement for 
program improvement. Specifically, the new rule (34 CFR 200.38) requires a school with 
a Chapter 1 program to conduct an annual review of the educational effectiveness of each 
school's project (Education Funding Research Council, 1990). If schools do not make 
substantial progress in both basic and advanced skills the schools are required to enter a 
program improvement process. 

Substantial progress is defined in terms of aggregate performance on a norm- 
referenced achievement test and other desired outcomes. Desired outcomes are defined 
as goal statements or measurable objectives which focus on what children will learn and 
accomplish as a result of their participation in the program. Desired outcomes can be 
measured by alternative assessment strategies different from standardized norm-referenced 
tests. In fact, the inclusion of desired outcomes as one method to assess substantial 
progress was in recognition of the position that norm-referenced tests often do not 
capture the growth made by Chapter 1 students. Thus, Chapter 1 promotes the use of 
alternative assessments. The types of alternative assessments that can be used to measure 
desired outcomes include promotion indicators (e.g., grades, retention rates), progress 
checks (e.g., number and types of problems solved), and performance-based assessments 
such as portfolio assessments. Although Chapter 1 promotes the use of alternative 
assessment procedures, there is little information about them, especially portfolios, to 
guide their implementation in Chapter 1; which factors facilitate implementation and 
which factors hamper implementation. 
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Schcx)ls are not only encouraged to use alternative forms of assessment to evaluate 
the effectiveness of their Chapter 1 program, but they can also use alternative assessment 
as part of a program improvement plan to enhance student learning. When schools are 
required to enter a program improvement process, the school must develop and 
implement plans to improve the performance of students in the program. Schools are 
encouraged to select from a number of different methods to improve student performance 
such as implementing new instructional strategies, improving the coordination between 
Chapter 1 and the regular classroom teachers, and monitoring student progress. Because 
of the renewed interest in alternative assessments, some schools which are in a program 
improvement process are developing portfolio assessment systems as a way to monitor 
student progress and to enhance student learning. While a dearth of studies exists on 
portfolios in general (Worthen, 1993) some advocate portfolios as a viable means to 
integrate instruction and assessment, encourage student success and keep better track of 
children's individual progress, thereby improving student performance. The 
implementation and impact of portfolio assessment systems developed in response to 
program improvement has not been investigated to support these claims. Thus, the 
purpose of the research was to document the implementation of a portfolio assessment 
system in response to mandated program improvement and assess its impact on teacher 
and student behaviors. Specifically, the research questions were: 

1. What organizational, logistic, and teacher*behavior factors are associated with 
the implementation of the portfolio assessment system and how do they 
influence this process? 

2. What effect does the portfolio assessment system have on teacher instructional 
strategies? 
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3. What effect does the portfolio assessment system have on coordination between 
Chapter 1 and regular classroom teachers? 

4. What effect does the portfolio assessment system have on student achievement 
and learning behavior? 

Methodology 

Participants in this study were elementary and middle school teachers and students 
from a rural school district in northern California. Three school buildings were 
represented. Since Chapter 1 students in these buildings had not made substantial 
progress in basic and advanced skills, the schools were identified for mandatory program 
improvement. As an integral part of their program improvement plan, the three schools 
decided to implement a portfolio assessment system to increase student learning. 

The portfolio project and research on it consists of two phases. The first phase 
was devoted to a pilot portfolio project which was implemented for the 1992-93 school 
year. Participants included one to three teachers per grade level (grades K - 8) in each of 
three school buildings. Each teacher worked with a selected number of students in their 
class for the pilot The total number of teachers participating during the pilot was 31. 
The purpose of the pilot was to work-out some of the management decisions about 
implementation and to determine if portfolio assessment was feasible at the district level. 
In terms of research activities, data collection focused primarily on question 1 for the first 
phase, and some information was collected on research question 2. 

The second phase deals with full implementation of the portfolio assessment 
system in all schools with all faculty and students participating. Timing for this component 
is to begin during the 1993 - 94 school year. Research activities for the second phase will 



focus on data collection on research questions 2, 3 and 4. Checks on interrater reliability 
will also be performed during this phase. 

The iritial tuii of the project began with three days of in-service training of 
teachers prrticipating in the pilot concerning portfolio, rubric, and standards development 
and monitoring. This was done in the context of a framework of sound assessment 
practice (Airasian, 1991). Also during the first phase, we made monthly on-site visits to 
monitor and support project implementation. 

During the first phase of the project and research, data were collected from 
teachers in three ways. First, the Stages of Concern (SoC) questionnaire based on the 
Concerns Based Adoption Model, or CBAM (Loucks and Hall, 1977; Parker and Griffin, 
1979), was administered during one of the monthly on-site visits after teachers had two 
months to begin to develop and use their portfolios. The data provided information on 
the concerns that teachers had about implementation of the portfolio system and further 
staff development needs. Second, Levels of Use (LoU) interviews, also from the CBAM 
framework, were conducted with a sample of teachers concerning implementation of the 
portfolio systems in their own classrooms. Hnally, a variety of information was collected 
during the monthly on site-visits: observations of teacher portfolios, semi-structured 
interviews with teachers, and minutes from the monthly meetings of the District Portfolio 
Committee meetings. The District Portfolio Committee was established to review and 
monitor portfolio implementation. 

Once the data was collected, the information was analyzed for the first-phase 
research questions. To this end, the survey and interview data were summarized, and 
systematically integrated with field notes from the monthly site visits. 



Results 

Before reporting the data collected to answer the research questions, it is 
important to describe exactly how far along teachers were in developing their pilot 
portfolio assessment program. To date, teachers and the District Portfolio Committee 
have made substantial progress in building a portfolio assessment system to determine if 
portfolios will help them better assess student progress, coordinate Chapter 1 with the 
regular classroom, and enhance student learning. All of the decisions that teachers made 
regarding their portfolio (goals, management, areas to be assessed, methods of assessment, 
and rubrics) were put into a brochure which they distribute to new teachers who become 
involved in the project. 

The first step teachers completed was to identify the goals and purpose of their 
portfolio program for the different populations; the district. Chapter 1 program, teachers, 
students and parents. Together, the goals and purposes for the different populations 
provided a direction for the project which is to monitor student progress to enhance 
student learning. Next, teachers decided what they wanted to assess as part of this pilot 
portfolio project. They decided to assess writing, reading comprehension, and oral 
language. Because these are rather broad categories of learning, teachers decided to focus 
on one specific component, tied to the district's curriculum, for each of the three broader 
categories. For writing, teachers focused on descriptive writing. For reading 
comprehension, they foca'^ed on literal comprehension including main idea, sequencing, 
following instructions and remembering facts. And for oral language, they focused on 
speaking and listening. 



After identifying what they thought was important to assess, teachers identiHed 
how to assess each area, and built rubrics for each assessment instrument. For descriptive 
writing, teachers planned to collect writing samples and developed a wholistic rubric to 
assess them. For literal comprehension, teachers decided to use different methods of 
assessment (eg., webbing, investigations), depending upon the grade level, and also 
developed a wholistic rubric to assess them. Teachers had yet to identify the assessment 
instrument or rubric for speaking and listening. 

In addition to the three assessment areas, teachers also wanted students to 
complete a self-reflection letter where students would explain the reason for selecting the 
pieces they put into their portfolios, and what they learned from the process. However, 
teachers had yet to work out the details for when students would complete these letters 
and, to date, no students had written self-reflection letters for their portfolios. 

Even though teachers had yet to complete the details for each assessment area, 
because of a lack of time, the district portfolio committee decided to proceed based on 
the areas they had completed in their design. They decided to focus their initial efforts on 
collecting and assessing writing samples. The main factor that prohibited teachen from 
completing the design of their portfolio assessment program was the time necessary to 
meet. Teachers hoped that the district would provide the necessary support to complete 
their portfolio design. In the hope of meeting this objective and gaining district 
administrative support, the District Portfolio Committee agreed, at the request of the 
school board, to present the project at a meeting of the school board in April 1993. 



Organizational, logistic and teacher-behavior factors associated with implementation of a 
portfolio system: 

Data to answer questions regarding the pilot phase of the project were collected 
from Stages of Concern questionnaires administered in November 1992, Levels of Use 
interviews conducted in March 1993, and site observations which were conducted monthly 
from September, 1992 to April, 1993 in conjunction with monthly District Portfolio 
Committee meetings. Results from these data collection activities are presented below. 
Stages of Concern. 

Educational innovations commonly take a period of time to be fully integrated in 
school settings, and depend on factors such as the level of training and support, as well as 
the project's complexity. At this point as what can be best characterized as an early stage 
in the adoption of the portfolio assessment system, most teachers are concerned with 
management issues as is common in the early stage of adopting an innovation. Although 
common, it is a very important stage of concern. If teachers do not resolve basic 
management issues, which focus on the effective integration of the innovation into their 
daily routine, they will drop the innovation. Teachers need to move the project to the 
point where it can be conducted in a routine, elective manner if the project is to survive. 

Figure 1 presents a summary of the Stages of Concern questionnaire completed by 
17 staff at the three sc:hools where each Stage of Concern is expressed as a percentile 
score; the higher the percentile score, the more intense the concern. 
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The goal of interpreting the SoC Proflles is to present an overall perspective of the 
relative intensity among the Stages of Concern. It is important to note that profiles 
represent the average concerns of teachers, and it doesn't necessarily mean that all 
teachers have the same relative intensity of concern for each stage. In fact individual 
patterns of concerns across stages can be quite different among teachers. 

The results indicate that most teachers have high Awareness, Informational, 
Personal and Management concerns. This pattern of responses is characteristic of new 
and non-users of an innovation, which is consistent with the fact that this is the first year 
in the implementation of the project. They are looking for additional information about 
the ^project and are concerned about how the project will impact what they do in the 
classroom, and how it will affect their time. They also have management concerns about 
portfolio implementation: they are interested in practical matters, specifically in terms of 
program logistics. Most teachers are concerned with how to effectively and efficiently 
integrate portfolios into their instructional program. Teachers seem to be equally 
concerned with how portfolios will be integrated into their current assessment system. No 
strong negative concerns about the innovation were identified. 

It is interesting to note that while there is a small peak at Collaboration concerns, 
it is not as high as the other concerns listed above. Collaboration concerns are 
characteristic of individuals who are implementing an innovation that requires 
collaboration with others who are within a person's sphere of influence. It could be that 
teachers have not arrived to the point that they are actively involved with other teachers 
about portfolios and how they can be implemented. Instead, teachers are more concerned 
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about trying to determine how best to implement portfolios in the classroom effectively 
and efficiently. 

The data indicate there is a need to train staff who enter the program after the 
program development phase is completed so that they have a background similar to the 
other teachers. In addition, the program would benefit if teachers were provided further 
clarification of their role in the program and answer any questions that teachers may have. 
Illustrations of these concerns will be described as themes from the monthly observations 
and meetings. 
Levels of Use: 

The results from the Levels of Use (LoU) interviews determined that the three 
staff interviewed are at Level III, Mechanical Use. This result is not surprising given the 
results from the Stages of Concern questionnaire which found the teachers to be primarily 
concerned with information and management issues. These teachers are using portfolios, 
but as we will see later, they are not using all the components and are using them in a 
stepwise fashion, resulting in disjointed and superficial use. Equally important, the 
teachers are not collecting information for the portfolios efficiently, and have not 
integrated the portfolio system into their instructional program. 

In addition to the eight Levels of Use, each level is further divided in terms of 
seven categories (knowledge, acquiring information, sharing, assessing, planning, status 
reporting and performing). These categories represent the key functions that users carry 
out when they use an innovation, and provide insight and additional information about the 
typical behaviors the three teachers are engaged in. Although there is slight variations in 
the typical behaviors the teachers are engaged in, there is a great deal of overlap. For the 



most part, these three teachers know the logistical information to begin to implement 
portfolios in their classrooms on a day-today basis, for the short-term, but have not 
reconciled implementation over the long-term. They are looking for information about 
how best to implement portfolios efficiently and trying out some different procedures; 
however, at this point there is little sharing among teachers about management and 
logistic issues. They are examining the use of portfolios, primarily in terms of 
management issues, but they are also concerned about its role in the district assessment 
system, and whether parents will participate and support portfolio use. Perhaps because 
they are unsure about the future of portfolio use in the district, there is little planning or 
time spent, aside form the District Portfolio Committee, for organizing teachers and 
managing portfolios at either the individual or school or district level. 

In terms of their own perception of where they arc in the implementation process, 
teachers report that logistic, time and management issues are the focus of most of their 
efforts regarding portfolio implementation. While they are managing portfolios with 
varying degrees of efficiency, they have yet to understand how I) they will arrive at the 
goal or purpose of the portfolio system which is to monitor and assess student progress, 2) 
portfolios impact their instructional program, and 3) portfolios help students get more 
involved in their own learning. 
Monthly Meetings/Observations: 

Data were collected from the monthly District Portfolio Committee meetings. The 
purpose of these meetings was to review the progress of the portfolio implementation and 
discuss any concerns or issues teachers had. In addition, the meetings allowed us a time to 
observe portfolio collections, ask teachers questions about implementation, and discuss any 
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concerns and issues they had. Field notes were taken during the monthly visitations for 
the committee meetings, observations, interviews, and discussions with staff. The field 
notes were reviewed, analyzed, categorized and coded. Data from the categories are used 
to clarify and better describe the concerns and use of the portfolios which were identified 
in the SoC questionnaires and LoU interviews. Several themes emerged from the analyses 
of these data. They are discussed below. 

District Support A key concern was whether the distnct*s administration would 
support teachers' effort to implement the portfolio assessment Initially, teachers were 
concerned whether the district would accept portfolios as a form of assessment of student 
learning. Because portfolios was an alternative assessment, not a norm-referenced test, 
teachers did not know what the district would think of this teacher-initiated project. 
While the question of district support continued to be a concern of teachers during the 
length of the research, teachers iater became more concerned about whether the district 
would actively support the portfolio implementation with resources for staff development. 
Teachers soon realized that in order for schools and the district to implement the project 
successfully, teachers needed additional time for staff development (such as in the use of 
the portfolio, impact on instruction, and involving parents), and time to manage the 
portfolio system. In addition, teachers also needed additional time to complete the initial 
portfolio design. Finally, because teachers saw the project as a long-term commitment, 
they wondered if district support would also be long-term. 

Replacing Grades. Another concern which emerged about the portfolio 
implementation was whether portfolio assessment would replace grades to assess student 
learning. In fact, this concern was mentioned most frequently by teachers. Teachers 
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repeatedly asked whether portfolios would either replace or be conducted in conjunction 
with grades. Teachers were often told by the principal who headed the District Portfolio 
Committee that because they were only in the pilot phase of the project, any discussion of 
portfolios replacing grades to assess student learning was premature and needed to wait 
until after the project was in operation for a period of time. However, the scenario where 
portfolios would replace grades was presented to teachers as a possibility. 

Time consuming* One of the major concerns that teachers had about 
implementing portfolios was that they found the project more time consuming than they 
initially thought. This concern had three components: portfolio development, portfolio 
management and staff development. However, teachers really had no knowledge about 
how long that process would take partly because they had no idea what steps needed to be 
completed to develop a portfolio system. In addition, they were very concerned about 
how much time it would take them to manage the system; however, again they had no 
knowledge about the steps involved in that process. Teachers were beginning to raise 
questions not only about the amount of time it was now taking them to manage the 
information that they were collecting in this pilot, but also about when the project moved 
to the next phase where the entire district became involved. Finally, they were also 
concerned about the time requirements for staff development. They soon began to realize 
that without district backing and support, it would be difficult, if not impossible, to 
complete what they started. In a related theme, teachers also began to state that while 
teachers can be told the procedures for using portfolios (what to collect, when to collect 
it, etc.), the process of learning how to conduct these procedures correctly for their 
classrooms and how to integrate it into what they were already doing was slow and filled 
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with trial and error. Although teachers had initially thought they could have their 
assessment system "up and running'* by the end of the school year, they soon set their 
sights towards more reasonable progress goals, especially since they did not yet have 
district support* 

Parental Support Teachers were also very concerned about what parents would 
think about portfolios. To teachers, parents would be the litmus test for portfolios 
implementation at the district level. Teachers were concerned whether parents would 
accept portfolios to assess student learning, and they felt they needed additional 
information about how best to share portfolios with parents* They were also concerned 
about when and how parents could be involved in their children's portfolio. They felt that 
if parents did not support portfolios, teachers would be unable to implement it at a district 
level. 

Effect on instructional program. Although teachers were told that implementing 
a portfolio system may impact how they conduct their instructional program, teachers did 
not realize how and to what degree this would happen. Early in the process of 
implementation, teachers discovered that implementing a portfolio system required 
changes in their instructional program* Teachers stated that the implementation of 
portfolios affected how they conducted their instructional program and how and when 
they interacted with students. For example, many teachers had to change the way they 
conducted their writing classes, since implementing portfolios required different 
procedures for writing than teachers conducted previously. Now teachers emphasized 
more of a writing workshop approach to writing instruction. In addition, portfolios caused 
them to meet more regularly with students about their writing to review student work and 
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to share expectations with students based on the rubrics which teachers developed. As a 
resuh, teachers saw that portfolios had an instructional value to students because students 
had a better idea of what was expected of them. 

Portfolios evolve over time. Another theme that began to emerge towards the end 
of this research period was that the portfolio design began to evolve. Although teachers 
designed the portfolio at the beginning of the project, they hadn*t realized that the design 
might change over the course of the implementation even though they were told 
otherwise. By the middle of the school year, teachers were already beginning to raise 
questions about the design and suggesting changes. For example, teachers originally 
decided to assess writing samples wholistically; however, they began to question whether a 
wholistic scoring system would give them the type of information they wanted about 
student progress. As a result, they started to discuss the option of developing an analytic 
trait method to assess writing samples. They also talked about focusing on another genre 
of writing rather than descriptive writing. So, teachers realized that the portfolio design 
might not look the same at the end than at the beginning of the project. 

More balanced view of assessment. One of the outcomes of being involved in 
developing a portfolio system is that some teachers began to have a more balanced view of 
assessment Many of the teachers had a fairly negative view of norm-reference tests 
(NRT) and thought portfolios held the answer to their concerns. In fact some teachers 
became involved in portfolios in reaction to NRTs. By the end of this research period, 
some of the teachers expressed that their view of NRTs and assessment changed. Even 
though they plan to continue with the portfolio program and refine it further, they 
realized that portfolios can only provide some of the infonnation needed by a district. 
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They felt that NRTs can also provide useful information for the district and for teachers. 
Thus, as a result of being involved in developing portfolios, some teachers claimed they 
arrived at a more balanced view of assessment rather than a view based on emotions and 
politics. 

Discussion 

Although much has been written concerning the reliability and validity of 
alternative assessment instruments, little had been written about the process of 
implementation, especially regarding the concerns and issues that teachers have about 
implementation. This case study was an attempt to begin to identify some of these 
concerns about implementing alternative assessment instruments, speciOcally in terms of 
portfolio assessment. Although this study is meant to be a long-term study, results are 
already beginning to emerge about some of the research questions. Specifically, we 
identified some of the organizational, logistic and teacher-behavior factors associated with 
portfolio implementation. Later, we hope to have additional information about the effect 
of portfolios on the instructional program, coordination between Chapter 1 and the 
regular classroom, and its impact on student learning. 

If Chapter 1 programs use portfolios as an alternative form of assessment to 
evaluate student learning or as a way to enhance student achievement to remove 
themselves from mandated program improvement, then teacher concerns become 
important. In order to implement a portfolio system effectively, projects will have to 
address these teacher concerns. In addition, these factors also have implications in terms 

15 



of the standards required of any measurement instrument used to assess student learning. 
These issues are discussed below. 

Offfmizatkmat, loffstic, ami teacher-behavior facton associate 
imptementadon. 

The case study identified several organizational, logistic, and teacher-behavior 
factors which are important to implement portfolios at a school or district level. Perhaps 
one of the key organizational factors that facilitate portfolio implementation, is district 
support. Without district support, school or teacher-initiated efforts to implement 
portfolio assessment systems will have a difficult time being implemented. District support 
means resources. District support is necessary in order to provide the time necessary that 
teachers believe they need for staff development on portfolio, rubric, and standards 
development and the time they need to manage a complex project such as a portfolio 
assessment system. 

A second organizational factor, which was already alluded to, is the need for staff 
development. The process of portfolio, rubric and standards development v ithin the 
context of sound assessment practice is complex and an area where teachers have little, if 
any training (Stiggins, 1985). Professionals versed in assessment and the integration of 
instruction and assessment must provide school and district staff with the inscrvicc training 
they need to develop a valid and reliable portfolio assessment system. Based on the 
concerns of staff in the case study, staff development must be on-going since a portfolio 
design takes time to develop. In addition, because of teacher turnover and because 
portfolio designs evolve, staff development must be on-going to keep abreast of personnel 
changes and changes in the portfolio system. 
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The third organizational factor that must be addressed is the role that portfolios in 
student assessment. Teachers were very concerned about whether portfolios would 
replace grades as the method to assess student progress. The question that teachers 
wanted the district to answer was will portfolios be used in li<:u of or in conjunction with 
grades? Teachers felt that this question needed to be resolved before they continued to 
commit their time to developing and implementing portfolios. They did not want to 
continue their efforts if portfolios had no place in assessing student learning. 

Apart from the question of portfolios' relationship to grades, there is a larger 
question of portfolios* role in the entire district assessment system, including grades, 
district testing, and Chapter 1 assessment. Given teachers' concerns over portfolios 
relationship with grades, it probably would not be long before others would also raise 
questions about portfolios' relationship to other aspects of the district's assessment 
program. 

Finally, a key player for portfolio implementation in this study who are not often 
mentioned in the literature as aa important stakeholder, but who teachers believed were 
very important to the success of the project, are parents. Without parent approval, 
teachers felt they would neither be able to implement portfolios nor have portfolios 
become an important enough part of student assessment to make their efforts worthwhile. 
Teachers thought parents would be an important factor in influencing the school board's 
position on portfolios. 

There are several logistical factors that will influence the implementation of a 
portfolio assessment system. Certainly one that is often cited in the portfolio literature, 
and perhaps the most important factor is that developing and maintaining portfolios is 
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time-consuming. First teachers found that it takes time to be trained in classroom 
assessment and time to develop a portfolio assessment system. Teachers also began to 
find, even though they were only implementing a portion of the portfolio system they had 
designed, that they spent alot of time collecting, sorting, filing, and storing portfolios. 

Another logistical factor that teachers found important for portfolio assessment 
was managing the portfolio system, such as collecting samples to go into a portfolio. 
Teachers also had concerns about what to take out of the portfolio at the end of the year, 
how the portfolio will be stored over time, and where the portfolio will be stored. These 
questions arose, not only because teachers were concerned about the amount of time it 
would take, but were also concerned about the complexity of the management process. It 
was apparent that teachers wanted more information and experience in dealing with the 
process of portfolio management. 

A third logistical factor that can influence portfolio implementation is rubric 
development. For example, teachers were already considering changes in how to assess 
the descriptive writing samples, from a wholisitc to an analytic trait model. Teachers were 
concerned that the wholistic assessment would not give them the information they wanted 
and they were concerned whether the wholistic scoring gave teachers enough information 
so writing samples would be assessed consistently. 

There is one key teacher-behavior factors that influenced the portfolio 
implementation at this time: the effect of portfolios, or any alternative assessment 
measure, on the classroom's instructional program. When assessment is tied to instruction, 
as with alternative assessment measures, including portfolios, then their implementation 
requires changes in the teachers instructional program. In this study, portfolio 
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implementation caused some teachers to change how they conducted writing workshops. 
Even though teachers nay be in favor of portfolios, they may change their minds about 
portfolio implementation after they realize that it also requires changes in iheir 
instructional program. Thus portfolio will require a new set of behaviors from teachers 
not only in terms of assessment, but also changes in their instructional program. 

As the development of the portfolio system progresses, especially as an assessment 
instrument, the need for validation will become critical. Acceptance and credibility of this 
relatively new and innovative approach for Chapter 1 assessment will depend on a 
systematic validation e^ort. Linn, Baker, and Dunbar (1990) have offered a framework 
for validating complex performance-based assessments. Although it would be premature 
to consider all the criteria offered for validation this early in implementation, the criteria 
cost and eflidency and generalizability specified by Linn, Baker and Dunbar have shown 
to be central to the current work. 

Cost and efCciency relates to the ease with which assessment data can be collected 
and the expense at doing so. Many teachers have shown concern for the efficiency aspect. 
As the portfolio assessment system moves to full implementation ways for efficient data 
collection and storage will have to be found if the system has a chance for fruition. And 
in a time of budget crises in several states, especially California, the cost will have to be 
kept at a minimum before acceptance from the local community can be obtained. 

The other criterion for validation becoming apparent at this early date in 
implementation is the issue of generalizability of the assessment data. In other words, 
how generalizablc are the findings to a larger domain of achievement. Part of the issue of 
generalizability is interratcr reliability. As a recent Rand study on the Vermont Portfolio 
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Assessment System has shown (Rand, 1992), interrater reliability suffered rendering the 
generaiizability of the assessment data suspect. The report identiHcd the need for clear 
criteria contained in the rubrics and training to enhance reliability and generaiizability. 
Several teachers in the current project have stressed concern for the clarity and accuracy 
of the rubrics. Refinement of the rubrics and staff development will be essential 
ingredients for the data to be generalizable and gain the full acceptance of the teaching 
staff and local community. 
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Figure 1 Stages of Concern: Portfolio Implementation 
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